(19) 




02) 



Europalsches Patentamt 
European Patent Office 
Office europeen das brevets 



(1D EP 1 158 473 A2 

EUROPEAN PATENT APPLICATION 



(43) 


Date of publication: 


(51) Intci7: G08B 13/193, G08G 1/16, 




28.1 1.2001 Bulletin 2001/48 




pnoo -i c/nn 
LaUotJ ib/UU 


(21) 


Application number 01304561.2 






(22) 


Date of fifing: 23.05.2001 






(84) 


Designated Contracting States: 


(72) 


Inventors: 




AT BE CH CY DE DK ES Ft FR GB GR IE IT U LU 


• 


Kumata, Klyoshi 




MC NL PTSETR 




Miyazu, Kyotanabe-shl, Kyoto (JP) 




Designated Extension States: 




Shlgota, Toru 




AL LT LV MK RO SI 




Nara-shl (JP) 


(30) 


Priority: 23.05.2000 JP 2000152208 


(74) 


Representative: Brown, Kenneth Richard et al 








R.G.C. Jenkins & Co. 


(71) 


Applicant: Sharp Kabushikl Kalsha 




26 Caxton Street 




Osaka-shl, Osaka 545-8522 (JP) 




London SW1H OR J (GB) 



(54) Surround surveillance system for mobile body, and mobile body, car, and train using the 
same 



(57) A surround surveillance system mounted on a 
mobile body for surveying surroundings around the mo- 
bile body includes an omniazimuth visual system, the 
omntazimuth visual system including: at least one om- 
niazimuth visual sensor including an optical system ca- 
pable of obtaining an image of 360° view field area 
the rea round and capable of central projection transfor- 
mation for the image, and an imaging section for con- 



verting the image obtained by the optical system into 
first image data; an image processor for transforming 
the first image data into second image data for a pano- 
ramic image and/or for a perspective image; a display 
section for displaying the panoramic image and/or the 
perspective image based on the second image data; 
and a display control section for selecting and control- 
ling the panoramic image and/or the perspective image. 



GO 



FIG.2 4a 



Irnoojng 
section 4b ' 



Optical 
system 




70- 
7-, 



Image 



/ 

200 



Switching 
section 



Display control 
section 



5b 



7a | 



Vehicle location 
detection section! 



—6b 



Display section 



V 



Alarm generation 
section 



a. 

UJ 



Ptbted by Jouve. 76001 R&fflS (FR) 




EP 1 158 473 A2 



Description 

BACKGROUND OF THE INVENTION 

J 1. FIELD OF THE INVENTION: 

[0001] The present invention relates to a surround surveillance system. In particular, the present invention relates 
to a surround surveillance system for a mobile body which is preferably used for surround surveillance of a car, a train, 
etc., for human and cargo transportation. Furthermore, the present invention relates to a mobile body (a car, a train, 
io etc.) which uses the surround surveillance system. 

2. DESCRIPTION OF THE RELATED ART: 

[0002] In recent years, an increase in traffic accidents has become a major social problem. In particular, in a crossroad 
'5 or the like, various accidents may sometimes occur. For example, people rush out into the street in which cars are 
travelling, a car collides head-on or into the rear of another car, etc. It is believed, in general, that such accidents are 
caused because a field of view for drivers and pedestrians is limited in the crossroad area, and many of the drivers 
and pedestrians do not pay attention to their surroundings and cannot quickly recognize dangers. Thus, improvement 
in a car itself, arousal of attention of drivers, improvement and maintenance of traffic environment, etc., are highly 
20 demanded. 

[0003] Conventionally, for the purpose of improving traffic environment, mirrors are installed at appropriate posh ions 
in a crossroad area such that the drivers and pedestrians can see blind areas behind obstacles. However, the amount 
of blind area which can be covered by a mirror is limited and, furthermore, a sufficient number of mirrors have not been 
installed. 

25 [0004] In recent years, many large motor vehicles, such as buses and some passenger cars, have a surveillance 
system for checking the safety therearound, especially at a rear side of the vehicle. The system includes a surveillance 
camera installed in the rear of the vehicle, and a monitor provided near a driver's seat or on a dashboard. The monitor 
is connected to the surveillance camera via a cable. An image obtained by the surveillance camera is displayed on 
the monitor However, even with such a surveillance system, the driver must check the safety at both sides of the 

30 vehicle mainly by his/her own eyes. Accordingly, in a crossroad area or the like, in which there are blind areas because 
of obstacles, the driver sometimes cannot quickly recognize dangers. Furthermore, a camera of this type has a limited 
field of view so that the camera can detect obstacles and anticipate the danger of collision only in one direction. In 
order to check the presence/absence of obstacles and anticipate the danger of collision over a wide range, a certain 
manipulation, e.g., alteration of a camera angle, is required. 

35 [0005] Since a primary purpose of the conventional surround surveillance system for motor vehicles is surveillance 
in one direction, a plurality of cameras are required for watching a 360° area around a motor vehicle; i.e. , it is necessary 
to provide four or more cameras such that each of front, rear, left, and right sides of the vehicle is provided with at least 
one camera. 

[0006] Also, the monitor of the surveillance system must be installed at a position such that the driver can easily see 
40 the screen of the monitor from the driver's seat at a frontal portion of the interior of the vehicle. Thus, positions at which 
the monitor can be installed are limited. 

[0007] In recent years, vehicle location display systems (car navigation systems) for displaying the position of a 
vehicle by utilizing a global positioning system (GPS) or the like have been widespread, and the number of cars which 
has a display device has been increasing. Thus, if a vehicle has a surveillance camera system and a car navigation 

4S system, a monitor of the surveillance camera system and a display device of the car navigation system occupy a large 
area and, hence, narrow the space around the driver's seat because they are separately provided. In many cases, ft 
is impossible to install both the monitor and the display device at a position such that the driver can easily see the 
screen of the monitor from the driver's seat Furthermore, it is troublesome to manipulate two systems at one time. 
[0008] As a matter of course, in the case of using a motor vehicle, a driver ts required to secure the safety around 

so the motor vehicle. For example, when the driver starts to drive, the driver has to check the safety at the right, left, and 
rear sides of the motor vehicle, as well as the front side. Naturally, when the motor vehicle turns right or (eft, or when 
the driver parks the motor vehicle in a carport or drives the vehicle out of the carport, the driver has to check the safety 
around the motor vehicle. However, due to the shape and structure of the vehicle, there are driver's blind areas, i.e., 
there are areas that the driver cannot see directly behind and/or around the vehicle, and ft is difficult for the driver to 

55 check the safety in the driver's blind areas. As a result, such blind areas impose a considerable burden on the driver. 
[0009] Furthermore, in the case of using a conventional surround surveillance system, it is necessary to provide a 
plurality of cameras for checking the safety in a 360° area around the vehicle. In such a case, the driver has to selectively 
switch the cameras from one to another, and/or turn the direction of the selected camera according to circumstances, 
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in order to check the safety around the vehicle. Such a manipulation is a considerable burden for the driver. 
SUMMARY OF THE INVENTION 

5 [0010] According to one aspect of the present invention, a surround surveillance system mounted on a mobile body 
for surveying surroundings around the mobile body includes an omniazimuth visual system, the omniazimuth visual 
system including: at least one omniazimuth visual sensor including an optical system capable of obtaining an image 
of 360° view field area there around and capable of central projection transformation for the image, and an imaging 
section for converting the image obtained by the optical system into first image data; an image processor for trans- 

'0 forming the first image data into second image data for a panoramic image and/or for a perspective image; a display 
section for displaying the panoramic image and/or the perspective image based on the second image data; and a 
display control section for selecting and controlling the panoramic image and/or the perspective image. 
[0011] In one embodiment of the present invention, the display section displays the panoramic image and the per- 
spective image at one time, or the display section selectively displays one of the panoramic image and the perspective 

ts image. 

[0012] In another embodiment of the present invention, the display section simultaneously displays at least frontal, 

left, and right view field perspective images within the 360° view field area based on the second image data. 

[0013] In still another embodiment of the present invention, the display control section selects one of the frontal, left, 

and right view field perspective images displayed by the display section; the image processor vertically /horizontally 
20 moves or scales-up/scales-down the view field perspective image selected by the display control section according to 

an external operation; and the display section displays the moved or scaled-up/scaled-down image. 

[0014] In still another embodiment of the present invention, the display section includes a location display section 

for displaying a mobile body location image; and the display control section switches the display section between an 

image showing surroundings of the mobile body and the mobile body location image. 
2S [0015] In still another embodiment of the present invention, the mobile body is a motor vehicle. 

[0016] In still another embodiment of the present invention, the at least one omniazimuth visual sensor is placed on 

a roof of the motor vehicle. 

[001 7] In still another embodiment of the present invention, the at least one omniazimuth visual sensor includes first 
and second omniazimuth visual sensors; the first omniazimuth visual sensor is placed on a front bumper of the motor 
so vehicle; and the second omniazimuth visual sensor is placed on a rear bumper of the motor vehicle. 

[0018] In stilt another embodiment of the present invention, the first omniazimuth visual sensor is placed on a left or 
right corner of the front bumper; and the second omniazimuth visual sensor is placed at a diagonal position on the rear 
bumper with respect to the first omniazimuth visual sensor. 

[0019] In still another embodiment of the present invention, the mobile body is a train. 

35 [0020] In sti It another embodi ment of the present invention, the surround surveillance system further includes: means 
for determining a distance between the mobile body and an object around the mobile body, a relative velocity of the 
object with respect to the mobile body, and a moving direction of the object based on a signal of the image data from 
the at least one omniazimuth visual sensor and a velocity signal from the mobile body; and alarming means for producing 
alarming information when the object comes into a predetermined area around the mobile body. 

•to [0021] According to another aspect of the present invention, a surround surveillance system includes: an omniazi- 
muth visual sensor including an optical system capable of obtaining an image of 360° view field area therearound and 
capable of central projection transformation for the image, and an imaging section for converting the image obtained 
by the optical system into first image data; an image processor for transforming the first image data into second image 
data for a panoramic image and/or for a perspective image; a display section for displaying the panoramic image and/ 

45 or the perspective image based on the second image data; and a display control section for selecting and controlling 
the panoramic image and/or the perspective image. 

[0022] According to still another aspect of the present invention, a mobile body includes the surround surveillance 
system according to the second aspect of the present invention. 

[0023] According to still another aspect of the present invention, a motor vehicle includes the surround surveillance 
so system according to the second aspect of the present invention. 

[0024] According to still another aspect of the present invention, a train includes the surround surveillance system 
according to the second aspect of the present invention. 

[0025] In the present specification, the phrase "an optical system is capable of central projection transformation" 
means that an imaging device is capable of acquiring an image which corresponds to an image seen from one of a 
ss plurality of focal points of an optical system. 

[0026] Hereinafter, functions of the present invention will be described. 

[0027] A surround surveillance system according to the present invention uses, as a part of an omniazimuth visual 
sensor, an optical system which is capable of obtaining an image of 360° view field area around a mobile body and 
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capable of central projection transformation for the image. An image obtained by such an optical system is converted 
into first image data by an imaging section, and the first image data is transformed into a panoramic or perspective 
image, thereby obtaining second image data. The second image data is displayed on the display section. Selection of 
image and the size of the selected image are controlled by the display selection section. With such a structure of the 
s present invention, a driver can check the safety around the mobile body without switching a plurality of cameras or 
changing the direction of the camera as in the conventional vehicle surveillance apparatus, the primary purpose of 
which is surveillance in one direction. 

[002B] For example, an omniazimuth visual sensor(s) is placed on a roof or on a front or rear bumper of an automobile, 
whereby driver's blind areas can be readily watched. Alternatively, the surround surveillance system according to the 

to present invention can be applied not only to automobiles but also to trains. 

[0029] The display section can display a panoramic image and a perspective image at one time, or selectively display 
one of the panoramic image and the perspective image. Alternatively, among frontal, rear, left, and right view field 
perspective images, the display section can display at least frontal, left, and right view field perspective images at one 
time. When necessary, the display section displays the rear view field perspective image. Furthermore, the display 

is control section may select one image, and the selected image may be vertically /horizontally moved (pan/tilt movement) 
or scale d-up/sc at ed- down by an image processor according to an external key operation. In this way, an image to be 
displayed can be selected, and the display direction and the size of the selected image can be freely selected/controlled. 
Thus, the driver can easily check the safety around the mobile body. 

[0030] The surround surveillance system further includes a location display section which displays the location of 
50 the mobile body (vehicle) on a map screen using a GPS or the like. The display control section enables the selective 
display of an image showing surroundings of the mobile body and a location display of the mobile body. With such an 
arrangement, the space around the driver's seat is not narrowed, and manipulation is not complicated; i.e., problems 
of the conventional system are avoided. 

[0031] The surround surveillance system further includes means for determining a distance from an object around 
25 the mobile body, the relative velocity of the mobile body, a moving direction of the mobile body, etc., which are deter- 
mined based on an image signal from the omniazimuth visual sensor and a velocity signal from the mobile body. The 
surround surveillance system further includes means for producing alarming information when the object comes into 
a predetermined distance area around the mobile body. With such an arrangement, a safety check can be readily 
performed. 

30 [0032] Thus, the invention described herein makes possible the advantages of (1) providing a surround surveillance 
system for readily observing surroundings of a mobile body in order to reduce a driver's burden and improve the safety 
around the mobile body and (2) providing a mobile body (a vehicle, a train, etc.) including the surround surveillance 
system. 

[0033] These and other advantages of the present invention will become apparent to those skilled in the art upon 
35 reading and understanding the following detailed description with reference to the accompanying figures. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0034] 

40 

Figure 1 A is a plan view showing a vehicle including a surround surveillance system for a mobile body according 
to embodiment 1 of the present invention. Figure 1B is a side view of the vehicle. 

Figure 2 is a block diagram showing a configuration of a surround surveillance system according to embodiment 1 . 

45 

Figure 3 shows a configuration example of an optical system according to embodiment 1 . 

Figure 4 is a block diagram showing a configuration example of the image processor 5. 

so Figure 5 is a block diagram showing a configuration example of an image transformation section 5a included in 

the image processor 5. 

Figure 6 is a block diagram showing a configuration example of an image comparison/distance determination 
section 5b included in the image processor S. 

55 

Figure 7 illustrates an example of panoramic (360°) image transformation according to embodiment 1 . Part (a) 
shows an input round-shape image. Part (b) shows a donut-shape image subjected to the panoramic image trans- 
formation. Part (c) shows a panoramic image obtained by transformation into a rectangular coordinate. 
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Figure 8 illustrates a perspective transformation according to embodiment 1 . 

Figure 9 is a schematic view for illustrating a principle of distance determination according to embodiment 1 . 

5 Figure 10 shows an example of a display screen 25 of the display section 6. 

Figure 11 A is a plan view showing a vehicle including a surround surveillance system for a mobile body according 
to embodiment 2 of the present invention. Figure 11 B is a side view of the vehicle. 



to Figure 12A is a plan view showing a vehicle including a surround surveillance system for a mobile body according 

to embodiment 3 of the present invention. Figure 12B is a side view of the vehicle. 



15 



Figure 13A is a side view showing a train which includes a surround surveillance system for a mobile body according 
to embodiment 4 of the present invention. Figure 13B is a plan view of the train 37 shown in Figure 13A. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 



[0035] Hereinafter, embodiments of the present invention will be described with reference to the drawings. 



20 (Embodiment 1) 

[0036] Figure 1 A is a plan view showing a vehicle 1 which includes a surround surveillance system for a mobile body 
according to embodiment 1 of the present invention. Figure 1 B is a side view of the vehicle 1 . The vehicle 1 has a front 
bumper 2. a rear bumper 3, and an omniazimuth visual sensor 4. 
25 [0037] In embodiment 1 , the omniazimuth visual sensor 4 is located on a roof of the vehicle 1 , and capable of obtaining 
an image of 360° view field area around the vehicle 1 in a generally horizontal direction. 

[0038] Figure 2 is a block diagram showing a configuration of a surround surveillance system 200 for use in a mobile 
body (vehicle 1), which is an example of an omniazimuth visual system according to embodiment 1 of the present 
invention. 

so [0039] The surround surveillance system 200 includes the omniazimuth visual sensor 4, an image processor 5, a 
display section 6, a display control section 7, an alarm generation section 8, and a vehicle location detection section 9. 
[0040] The omniazimuth visual sensor 4 includes an optical system 4a capable of obtaining an image of 360° view 
field area therearound and capable of central projection transformation for the image, and an imaging section 4b for 
converting the image obtained by the optical system 4a into image data. 

35 [0041] The image processor 5 includes: an image transformation section 5a for transforming the image data obtained 
by the imaging section 4b into a panoramic image, a perspective image, etc.; an image comparison/distance determi- 
nation section 5b for detecting an object around the omniazimuth visual sensor 4 by comparing image data obtained 
at different times with a predetermined time period therebetween, and for determining the distance from the object, 
the relative velocity with respect to the object, the moving direction of the object, etc., based on the displacement of 

40 the object between the different image data and a velocity signal from the omniazimuth visual sensor 4 which represents 
the speed of the vehicle 1; and an output buffer memory 5c. 

[0042] The vehicle location detection section 9 detects a location of a vehicle in which it is installed (i.e., the location 
of the vehicle 1) in a map displayed on the display section 6 using the GPS or the like. The display section 6 can 
selectively display an output 6a of the image processor 5 and an output 6b of the vehicle location detection section 9. 
45 [0043] The display control section 7 controls the selection among images of surroundings of the vehicle and the size 
of the selected image. Furthermore, the display control section 7 outputs to the display section 6 a control signal 7a 
for controlling a switch between the image of the surrounding of the vehicle 1 (the omniazimuth visual sensor 4) and 
the vehicle location image. 

[0044] The alarm generation section 8 generates alarm information when an object comes into a predetermined area 
so around the vehicle 1. 

[0045] The display section 6 is placed in a position such that the driver can easily see the screen of the display 
section 6 and easily manipulate the display section 6. Preferably, the display section 6 is placed at a position on a front 
dashboard near the driver's seat such that the display section 6 does not narrow a frontal field of view of the driver, 
and the driver in the driver's seat can readily access the display section 6. The other components (the display processor 
55 5, the display control section 7, the alarm generation section 8, and the vehicle location detection section 9) are pref- 
erably placed in a zone in which temperature variation and vb ration are small. For example, in the case where they 
are placed in a luggage compartment (trunk compartment) at the rear end of the vehicle, it is preferable that they be 
placed at a possible distant position from an engine. 
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[0046] Each of these components is now described in detail with reference to the drawings. 

[0047] Figure 3 shows an example of the optical system 4a capable of central projection transformation. This optical 
system uses a hyperboloidal mirror 22 which has a shape of one sheet of a two-sheeted hyperboloid, which is an 
example of a mirror having a shape of a surface of revolution. The rotation axis of the hyperboloidal mirror 22 is identical 

3 with the optical axis of an imaging lens included in the imaging section 4b, and the first principal point of the imaging 
lens is located at one of focal points of the hyperboloidal mirror 22 (external focal point © ). In such a structure, an 
image obtained by the imaging section 4b corresponds to an image seen from the internal focal point © of the hyper- 
bolotdal mirror 22. Such an optical system is disclosed in. for example, Japanese Laid-Open Publication No. 6-295333, 
and only several features of the optical system are herein described. 

10 [0048] In Figure 3, the hyperboloidal mirror 22 is formed by providing a mirror on a convex surface of a body defined 
by one of curved surfaces obtained by rotating hyperbolic curves around a z-axis (two-sheeted hyperboloid). i.e., a 
region of the two-sheeted hyperboloid where Z>0. This two-sheeted hyperboloid is represented as: 



20 where a and b are constants for defining a shape of the hyperboloid, and c is a constant for defining a focal point of 
the hyperboloid. Hereinafter, the constants a, b, and c are gen erica Ify referred to as "mirror constants". 
[0049] The hyperboloidal mirror 22 has two focal points © and® • All of light from outside which travels toward 
focal point (D is reflected by the hyperboloidal mirror 22 so as to reach focal point @ - The hyperboloidal mirror 22 
and the imaging section 4b are positioned such that the rotation axis of the hyperboloidal mirror 22 is identical with the 

25 optical axis of an imaging lens of the imaging section 4b, and the first principal point of the imaging tens is located at 
focal point® . With such a configuration, an image obtained by the imaging section 4b corresponds to an image seen 
from focal point CD of the hyperboloidal mirror 22. 

[0050] The imaging section 4b may be a video camera or the like. The imaging section 4b converts an optical image 
obtained through the hyperboloidal mirror 22 of Figure 3 into image data using a solid-state imaging device, such as 
30 CCD, CMOS, etc. The converted image data is input to a first input buffer memory 11 of the image processor 5 (see 
Figure 4). A lens of the imaging section 4b may be a commonly-employed spherical lens or aspherical lens so long as 
the first principal point of the lens is located at focal point® • 

[0051] Figure 4 is a block diagram showing a configuration example of the image processor 5. Figure 5 is a block 
diagram showing a configuration example of an image transformation section 5a included in the image processor 5. 
35 Figure 6 is a block diagram showing a configuration example of an image comparison/distance determination section 
5b included in the image processor 5. 

[0052] As shown in Figures 4 and 5, the image transformation section 5a of the image processor 5 includes an A/D 
converter 10, a first input buffer memory 11, a CPU 12, a lookup table (LUT) 13, and an image transformation logic 14. 
[0053] As shown in Figures 4 and 6, the image comparison/distance determination section 5b of the image processor 
40 5 shares with the image transformation section 5a the A/0 converter 10, the first input buffer memory 1 1 , the CPU 12, 
the lookup table (LUT) 13, and further includes an image comparison/distance determination logic 16, a second input 
buffer memory 17, and a delay circuit 18. 

[0054] The output buffer memory 5c (Figure 4) of the image processor 5 is connected to each of the above compo- 
nents via a bus line 43. 

45 [0055] The image processor 5 receives image data from the imaging section 4b. When the image data is an analog 
signal, the analog signal is converted by the A/D converter 10 into a digital signal, and the digital signal is transmitted 
to the first input buffer memory 11 and further transmitted from the first input buffer memory 11 through the delay circuit 
18 to the second input buffer memory 17. When the image data is a digital signal, the image data is directly transmitted 
to the first input buffer memory 11 and transmitted through the delay circuit 18 to the second input buffer memory 17. 

50 [0056] In the image transformation section 5a of the image processor 5, the image transformation logic 14 processes 
an output (image data) of the first input buffer memory 11 using the lookup table (LUT) 13 so as to obtain a panoramic 
or perspective image, or so as to vertically/horizontally move or scale- up/scale-down an image. The image transfor- 
mation logic 14 performs other image processing when necessary. After the image transformation processing, the 
processed image data rs input to the output buffer .memory 5c During the processing, the components are controlled 

55 by the CPU 12. If the CPU 12 has a parallel processing function, faster processing speed is achieved. 

[0057] A principle of the image transformation by the image transformation logic 14 is now described. The image 
transformation includes a panoramic transformation for obtaining a panoramic (360°) image and a perspective trans- 
formation for obtaining a perspective image. Furthermore, the perspective transformation includes a horizontally rata- 
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tional transfer (horizontal transfer, so-called "pan movement") and a vertical!/ rotational transfer (vertical transfer, so- 
called "tilt movement"). 

[0058] First, a panoramic (360°) image transformation is described with reference to Figure 7. Referring to part (a) 
of Figure 7, an image 19 is a round-shape image obtained by the imaging section 4b. Part (b) of Figure 7 shows a 
S donut-shape image 20 subjected to the panoramic image transformation. Part (c) of Figure 7 shows a panoramic image 
21 obtained by transforming the image 19 into a rectangular coordinate. 

[0059] Part (a) of Figure 7 shows the input round-shape image 19 which is formatted in a polar coordinate form in 
which the center point of the image 19 is positioned at the origin (Xo.Yo) of the coordinates. In this polar coordinate, 
a pixel P in the image 19 is represented as P(r,6). Referring to part (c) of Figure 7, in the panoramic image 21, a point 
to corresponding to the pixel P in the image 19 (part (a) of Figure 7) can be represented as P(x,y). When the round-shape 
image 1 9 shown in part (a) of Figure 7 is transformed into the square panoramic image 21 shown in part (c) of Figure 
7 using a point PO(ro,0o) as a reference point, this transformation is represented by the following expressions: 

«c x = e-eo 



20 When the input round-shape image 19 (part (a) of Figure 7) is formatted into a rectangular coordinate such that the 
center point of the round-shape image 1 9 is positioned at the origin of the rectangular coordinate system, (Xo.Yo), the 
point P on the image 19 is represented as (X.Y). Accordingly. X and Y are represented as: 

„ X = Xo+rxcos9 

ZD 

Y- Yo+rXsinO 

so Thus, 

X - Xo+(y+ro)xcos(x+eo) 

35 Y = Yo+(y+ro)X sin(x+6o) 

[0060] In the pan movement for a panoramic image, a point obtained by increasing or decreasing M 6o N of the reference 
point PO(ro.Oo) by a certain angle 0 according to a predetermined key operation is used as a new reference point for 
40 the pan movement. With this new reference point for the pan movement, a horizontally panned panoramic image can 
be directly obtained from the input round-shape image 19. It should be noted that a tilt movement is not performed for 
a panoramic image. 

[0061] Next, a perspective transformation is described with reference to Figure 8l In the perspective transformation, 
the position of a point on the input image obtained by a light receiving section 4c of the imaging section 4b which 
45 corresponds to a point in a three-dimensional space is calculated, and image information at the point on the input 
image is allocated to a corresponding point on a perspective-transformed image, whereby coordinate transformation 
is performed. 

[0062] In particular, as shown in Figure 8, a point in a three-dimensional space is represented as P(tx,ty,tz), a point 
corresponding thereto which is on a round- shape image formed on a light receiving plane of a light receiving section 
50 4c of the imaging section 4b is represented as R(r,6), the focal distance of the light receiving section 4c of the imaging 
section 4b (a distance between a principal point of a lens and a receiving element of the light receiving section 4c) is 
F, and mirror constants are (a, b. c), which are the same as a, b, and c in Figure 3. With these parameters, expression 
(1 ) is obtained: 

55 r«Fxtan((x/2)-p) (1) 

In Figure 8, a is an incident angle of light which travels from an object point (point P) toward focal point © with respect 
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to a horizontal plane including focal point ® ; p is an incident angle of light which comes from point P. is reflected at 
point G on the hyperboloidal mirror 22, and enters into the imaging section Ob (angle between the incident light and a 
plane perpendicular to an optical axis of the light receiving section 4c of the imaging section 4b). Algebraic numbers 
a, p, and 9 are represented as follows: 

P = arctan(((b 2 +c 2 ) x sina-2 xbx c)/(b 2 -c 2 ) x cosa) 



10 a - arctan(tz/sqrt(tx 2 +ty 2 )) 

9 = arctan(tyAx) 

is From the above, expression (1) is represented as follows: 

r ** F x (((b 2 -c 2 ) x sqrt(tx 2 +ty 2 ))/<(b 2 +c 2 ) 
20 x tz-2 x b x c x sqrt(tx 2 +ty 2 +tz 2 ))) 

The coordinate of a point on the round-shape image is transformed into a rectangular coordinate P (X.Y). X and Y are 
represented as: 

25 

X« rxcosG 
Y = rxsin9 



Accordingly, from the above expressions: 

X- Fx (((b 2 -c 2 ) xtx/((b 2 +c 2 ) xtz-2 X b x c 

xsqrt(txW+tz 2 ))) (2) 

Y= Fx((<b 2 -c 2 )Xty/((b 2 +c 2 )Xtz-2XbXc 

Xsqrt(tx 2 +ty 2 -Hz 2 ))) (3) 

[0063] With the above expressions, object point P (tx,ty,tz) is perspectivety transformed onto the rectangular coor- 
dinate system. 

[0064] Now, referring to Figure 8, consider a square image plane having width W and height h and located in the 
three-dimensional space at a position corresponding to a rotation angle 0 around the Z-axis where R is a distance 
between the plane and focal point® of the hyperboloidal mirror 22, and $ is a depression angle (which is equal to 
the incident angle a). Parameters of a point at the upper left corner of the square image plane, point Q (txq,tyq,tzq), 
are represented as follows: 

txq = (Rcos*+(h/2)sin(>)cosO-(W/2)sine (4) 



tyq =» (Rcos$+(h/2)sin+)sin9+(W/2)cose (5) 



a 
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tzq * Rsin$-{h/2)cos$ (6) 

By combining expressions (4), (5), and (6) into expressions (2) and (3), it is possible to obtain the coordinate (X,Y) of 
5 a point on the round-shape image formed on the light receiving section 4c of the imaging section 4b which corresponds 
to point O of the square image plane. Furthermore, assume that the square image plane is transformed into a per- 
spective image divided into pixels each having a width d and a height e. In expressions (4), (5), and (6), the parameter 
W is changed in a range from W to -W on the units of W/d, and the parameter h is changed in a range from h to -h on 
the units of h/e, whereby coordinates of points on the square image plane are obtained. According to these obtained 
10 coordinates of the points on the square image plane, image data at points on the round -shape image formed on the 
light receiving section 4c which correspond to the points on the square image plane is transferred onto a perspective 
image. 

[0065] Next, a horizontally rotational movement (pan movement) and a vertically rotational movement (tilt movement) 
in the perspective transformation are described. First, a case where point P as mentioned above is horizontally and 
'5 rotationalty moved (pan movement) is descrbed. A coordinate of a point obtained after the horizontally rotational move- 
ment, point P (tx'.ty'.tz*), is represented as follows: 

U' = (Rcos<|»+(h/2)sin4»)cos(e+A0)-(W/2)sin(O+Ae) (7) 

20 

ty* = (Rce*$+(h/2)sin$)stn(9+Ae)+(W/2)cos{0+A9) (8) 

25 tz* = RsinHh/2)cos$ (9) 

where AG denotes a horizontal movement angle. 

[0066] By combining expressions (7), (8), and (9) into expressions (2) and (3), the coordinate (X.Y) of a point on the 
round-shape image formed on the light receiving section 4c which corresponds to the point P (tx'.ty'.tz*) can be obtained. 

so This applies to other points on the round-shape image. In expressions (7), (8), and (9), the parameter W is changed 
in a range from W to - W on the units of W/d, and the parameter h is changed in a range from h to -h on the units of 
h/e, whereby coordinates of points on the square image plane are obtained. According to these obtained coordinates 
of the points on the square image plane, image data at points on the round-shape image formed on the light receiving 
section 4c which correspond to the point P'ttx'.ty'.tz') is transferred onto a perspective image, whereby a horizontally 

35 rotated image can be obtained. 

[0067] Next, a case where point P as mentioned above is vertically and rotationally moved (tilt movement) is de- 
scribed. A coordinate of a point obtained after the vertically rotational movement, point P" (tx",ty",tz"), is represented 
as follows: 

40 tx tt - (Rcosf^+A^J+^sin^+A^JXcose-OW/SJsine (10) 

ty" =* (Rcos(4.+A4)+(h/2)sin(++A^)XsinG+(W/2)cos8 (11) 

45 

tz- = Rsin(4»+A4.)-(h/2)cos($+A4) (1 2) 

where A$ denotes a vertical movement angle. 

50 [0068] By combining expressions (10), (11 ), and (1 2) into expressions (2) and (3), the coordinate (X.Y) of a point on 
the round-shape image formed on the light receiving section 4c which corresponds to the point P" (tx\ty\tz") can be 
obtained. This applies to other points on the round-shape image. In expressions (1 0), (1 1 ), and (12), the parameter W 
is changed in a range from W to -W on the units of W/d, and the parameter h is changed in a range from h to -h on the 
units of h/e, whereby coordinates of points on the square image plane are obtained. According to these obtained 

55 coordinates of the points on the square image plane, image data at points on the round-shape image formed on the 
light receiving section 4c which correspond to the point P* (tx N ,ty",tz") is transferred onto a perspective image, whereby 
a vertically rotated image can be obtained. 

[0069] Further, a zoom-in/zoom-out function for a perspective image is achieved by one parameter, the parameter 
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R. In particular, tha parameter R in expressions (4) through (12) is changed by a certain amount aR according to a 
certain key operation, whereby a zoom -in/zoom -out image is generated directly from the round-shape input image 
formed on the light receiving section 4c. 

[0070] Furthermore, a transformation region determination function is achieved such that the range of a transforma- 

5 tion region in a radius direction of the round-shape input image farmed on the light receiving section 4c is determined 
by a certain key operation during the transformation from the round-shape input image into a panoramic image. When 
the imaging section is in a transformation region determination mode, a transformation region can be determined by 
a certain key operation. In particular, a transformation region in the round-shape input image is defined by two circles, 
i.e., as shown in part (a) of Figure 7, an inner circle including the reference point O(ro,0o) whose radius is ro and an 

10 outer circle which corresponds to an upper side of the panoramic image 21 shown in part (c) of Figure 7. The maximum 
radius of the round-shape input image formed on the light receiving section 4c is rmax, and the minimum radius of an 
image of the light receiving section 4c is rmin. The radiuses of the above two circles which define the transformation 
region can be freely determined within the range from rmin to rmax by a certain key operation. 
[0071] In the image comparison/distance determination section 5b shown in Figure 6, the image comparison/distance 

*5 determination logic 16 compares data stored in the first input buffer memory 11 and data stored in the second input 
buffer memory 17 so as to obtain angle data with respect to a target object, the velocity information which represents 
the speed of the vehicle 1, and a time difference between the data stored in the first input buffer memory 11 and the 
data stored in the second input buffer memory 17. From these obtained information, the image comparison/distance 
determination logic 16 calculates a distance between the vehicle 1 and the target object. 

20 [0072] A principle of the distance determination between the vehicle 1 and the target object is now described with 
reference to Figure 9. Part (a) of Figure 9 shows an input image 23 obtained at time tO and stored in the second input 
buffer memory 17. Part (b) of Figure 9 shows an input image 24 obtained t seconds after time tO and stored in the first 
input buffer memory 11. It is due to the delay circuit 18 (Figure 6) that the time (time tO) of the input image 23 stored 
in the second input buffer memory 17 and the time (time tO+t) of the input image 24 stored in the first input buffer 

25 memory 11 are different 

[0073] Image information obtained by the imaging section 4b at time tO is input to the first input buffer memory 11. 
The image information obtained at time tO is transmitted through the delay circuit 18 and reaches the second input 
buffer memory 17 1 seconds after the imaging section 4b is input to the first input buffer memory 1 1. At the time when 
the image information obtained at time tO is input to the second input buffer memory 17, image information obtained t 

so seconds after time tO is input to the first input buffer memory 11. Therefore, by comparing the data stored in the first 
input buffer memory 11 and the data stored in the second input buffer memory 17, a comparison can be made between 
the input image obtained at time tO and the input image obtained t seconds after time tO. 

[0074] In Part (a) of Figure 9, at time tO, an object A and an object B are at position (r1 ,91) and position (r2,y1) on 
the input image 23, respectively. In Part (b) of Figure 9, t seconds after time tO, the object A and the object B are at 
35 position (R1 ,82) and position (R2,y2) on the input image 24, respectively. 

[0075] A distance L that the vehicle 1 moved for t seconds is obtained as follows based on velocity information from 
a velocimeter of the vehicle 1 : 



where v denotes the velocity. (In this example, velocity v is constant for t seconds.) Thus, with the above two types of 
image information, the image comparison/distance determination logic 16 can calculate a distance between the vehicle 
1 and a target object based on the principle of triangulation. For example, t seconds after time tO, a distance La between 
45 the vehicle 1 and the object A and a distance Lb between the vehicle 1 and the object B are obtained as follows: 



Calculation results for La and Lb are sent to the display section 6 (Figure 2) and displayed thereon. Furthermore, when 
the object comes into a predetermined area around the vehicle 1 , the image processor 5 (Figure 2) outputs an alarming 
55 signal to the alarm generation section 8 (Figure 2) including a speaker, etc., and the alarm generation section 8 gives 
forth a warning sound. Meanwhile, referring to Figure 2, the alarming signal is also transmitted from the image processor 
5 to the display control section 7, and the display control section 7 produces an alarming display on a screen of the 
display section 6 so that, for example, a screen display of a perspective image flickers. In Figures 2 and 4, an output 



La= L01/(02-01) 



50 



Q3-L V 1/( V 2- V 1) 



10 




EP1 158 473 A2 

16a of the image comparison/distance determination logic 16 is an alarming signal to the alarm generation section 8, 
and an output 1 6b ol the image comparison/distance determination logic 16 is an alarming signal to the display control 
section 7. 

[0076] The display section 6 may be a monitor, or the like, of a cathode-ray tube, LCD, EL, etc. The display section 
5 6 receives an output from the output buffer memory 5c of the image processor 5 and displays an image. Under the 
control of the display control section 7, the display section 6 can display a panoramic image and a perspective image 
at one time, or selectively display one of the panoramic image and the perspective image. Furthermore, in the case of 
displaying the perspective image, the display section 6 displays a frontal view field perspective image and left and right 
view field perspective images at one time. Additionally, a rear view field perspective image can be displayed when 
10 necessary. Further still, the display control section 7 may select one of these perspective images, and the selected 
perspective image may be vertically/horizontally moved or scaled- up/scaled-down before H is displayed on the display 
section 6. 

[0077] Moreover, in response to a signal from a switching section 70 located on a front dashboard near the driver's 
seat, the display control section 7 switches a display on the screen of the display section 6 between a display of an 

'5 image showing surroundings of the vehicle 1 and a display of a vehicle location image. For exam pie, when the switching 
section directs the display control section 7 to display the vehicle location image, the display control section 7 displays 
vehicle location information obtained by the vehicle location detection section 9, such as a GPS or the like, on the 
display section 6. When the switching section directs the display control section 7 to display the image showing sur- 
roundings of the vehicle 1, the display control section 7 sends vehicle surround image information from the image 

20 processor 5 to the display section 6, and an image showing surroundings of the vehicle 1 is displayed on the display 
section 6 based on the vehicle surround image information. 

[0078] The display control section 7 may be a special-purpose microcomputer or the like. The display control section 
7 selects the type of an image to be displayed on the display section 6 (for example, a panoramic image, a perspective 
image, etc., obtained by the image transformation in the image processor 5), and controls the orientation and the size 
25 of the image. 

[0079] Figure 1 0 shows an example of a display screen 25 of the display section 6. The display screen 25 includes: 
a first perspective image display window 26 (in the default state, the first perspective image display window 26 displays 
a frontal view field perspective image); a first explanation display window 27 for showing an explanation of the first 
perspective image display window 26; a second perspective image display window 28 (in the default state, the second 

30 perspective image display window 28 displays a left view field perspective image); a second explanation display window 
29 for showing an explanation of the second perspective image display window 28; a third perspective image display 
window 30 (in the default state, the third perspective image display window 30 displays a right view field perspective 
image); a third explanation display window 31 for showing an explanation of the third perspective image display window 
30; a panoramic image display window 32 (in this example, a 360° image is shown); a fourth explanation display 

35 window 33 for showing an explanation of the panoramic image display window 32; a direction key 34 for vertically/ 
horizontally scrolling images; a scale-up key 35 for scaling up images; and a scale-down key 36 for scaling down 
images. 

[0080] The first through fourth explanation display windows 27, 29, 31 , and 33 function as switches for activating the 
image display windows 26, 28, 30, and 32. A user (driver) activates a desired image display window (window 26, 28, 

40 30, or 32) by means of a corresponding explanation display window (window 27, 29, 31, or 33) which functions as a 
switch, whereby the corresponding explanation display window changes its own display color, and the user can verti- 
cally/horizontally scroll and scale-up/down the image displayed in the activated window using the direction key 34, the 
scale-up key 35, and the scale-down key 36. It should be noted that an image displayed in the panoramic image display 
window 32 is not scaled-up or scaled-down. 

45 [0081] For example, when the user (driver) touches the first explanation display window 27, a signal is output to the 
display control section 7 (Figure 2). In response to the touch, the display control section 7 changes the display color 
of the first explanation display window 27 into a color which indicates the first perspective image display window 26 is 
active, or allows the first explanation display window 2 7 to flicker. Meanwhile, the first perspective image display window 
26 becomes active, and the user can vertically /horizontally scroll and scale-up/down the image displayed in the window 

so 26 using the direction key 34, the scale-up key 35, and the scale-down key 36. In particular, signals are sent from the 
direction key 34, the scale-up key 35, and the scale-down key 36 through the display control section 7 to the image 
transformation section 5a of the image processor 5 (Figure 2). According to the signals from the direction key 34, the 
scale-up key 35, and the scale-down key 36, an image is transformed, and the transformed image is transmitted to 
the display section 6 (Figure 2) and displayed on the screen 25 of the display section 6. 

55 

(Embodiment 2) 

[0082] Figure 11 A ts a plan view showing a vehicle 1 which includes a surround surveillance system for a mobile 



11 




EP 1 158 473 A2 

body according to embodiment 2 of the present invention. Figure 11B is a side view of the vehicle 1. 
[0083] In embodiment 2, the vehicle 1 has a front bumper 2, a rear bumper 3, and omniazimuth visual sensors 4. 
One of the omniazimuth visual sensors 4 is placed on the central portion of the front bumper 2, and the other is placed 
on the central portion of the rear bumper 3. Each of the omniazimuth visual sensor 4 has a 360° view field around itself 

5 in a generally horizontal direction. 

[0084] However, a half of the view field (rear view field) of the omniazimuth visual sensor 4 on the front bumper 2 is 
blocked by the vehicle 1 . That is, the view field of the omniazimuth visual sensor 4 is limited to the 180' frontal view 
field (from the left side to the right side of the vehicle 1). Similarly, a half of the view field (frontal view field) of the 
omniazimuth visual sensor 4 on the rear bumper 3 is blocked by the vehicle 1. That is, the view field of the omniazimuth 

10 visual sensor 4 is limited to the 180° rear view field (from the left side to the right side of the vehicle 1). Thus, with 
these two omniazimuth visual sensors 4, a view field of about 360° in total can be obtained. 

[0085] According to embodiment 1 , as shown in Figures 1 A and 1B, the omniazimuth visual sensor 4 is located on 
a roof of the vehicle 1. From such a location, one omniazimuth visual sensor 4 can obtain an image of 360° view field 
area around itself in a generally horizontal direction. However, as seen from Figures 1 A and 1 B t the omniazimuth visual 

'5 sensor 4 placed in such a location cannot see blind areas blocked by the roof: i.e., the omniazimuth visual sensor 4 
located on the roof of the vehicle 1 (embodiment 1) cannot see blind areas as close proximity to the vehicle 1 as the 
omniazimuth visual sensor 4 placed at the front and rear of the vehicle 1 (embodiment 2). Moreover, in a crossroad 
area where there are driver's blind areas behind obstacles at left-hand and right-hand sides of the vehicle 1, the vehicle 
1 should advance into the crossroad so that the omniazimuth visual sensor 4 can see the blind areas. On the other 

20 hand, according to embodiment 2, since the omniazimuth visual sensors 4 are respectively placed at the front and rear 
of the vehicle 1 , one of the omniazimuth visual sensors 4 can see the blind areas before the vehicle 1 deeply advances 
into the crossroad to such an extent that the vehicle 1 according to embodiment 1 does. Furthermore, since the view 
- fields of the omniazimuth visual sensors 4 are not blocked by the roof of the vehicle 1, the omniazimuth visual sensors 
4 can see areas in close proximity to the vehicle 1 at the front and rear sides. 

25 

(Embodiment 3) 

[0085] Figure 12A is a plan view showing a vehicle 1 which includes a surround surveillance system for a mobile 
body according to embodiment 3 of the present invention. Figure 12B is a side view of the vehicle 1. 
£0 [0087] According to embodiment 3, one of the omniazimuth visual sensors 4 is placed on the left corner of the front 
bumper 2, and the other is placed on the right corner of the rear bumper 3. Each of the omniazimuth visual sensors 4 
has a 360° view field around itself in a generally horizontal direction. 

[0088] However, one fourth of the view field (a right-hand half of the rear view field (about 90°)) of the omniazimuth 
visual sensor 4 on the front bumper 2 is blocked by the vehicle 1. That is, the view field of the omniazimuth visual 

3S sensor 4 is limited to about 270° front view field. Similarly, one fourth of the view field (a left-hand half of the front view 
field (about 90°)) of the omniazimuth visual sensor 4 on the rear bumper 3 is blocked by the vehicle 1. That is, the view 
field of the omniazimuth visual sensor 4 is limited to about 270° rear view field. Thus, with these two omniazimuth 
visual sensors 4, a view field of about 360° can be obtained such that the omniazimuth visual sensors 4 can see areas 
in cbse proximity to the vehicle 1 which are the blind areas of the vehicle 1 according to embodiment 1. 

40 [0089] Also in embodiment 3, in a crossroad area where there are driver's blind areas behind obstacles at left-hand 
and right-hand sides of the vehicle 1, the vehicle 1 does not need to deeply advance into the crossroad so as to see 
the blind areas at right and left sides. Furthermore, since the view fields of the omniazimuth visual sensors 4 are not 
blocked by the roof of the vehicle 1 as in embodiment 1, the omniazimuth visual sensors 4 can see areas in close 
proximity to the vehicle 1 at the front, rear, left, and right sides thereof. 

45 [0090] In embodiments 1-3, the vehicle 1 shown in the drawings is an automobile for passengers. However, the 
present invention also can be applied to a targe vehicle, such as a bus or the like, and a vehicle for cargoes. In particu tar, 
the present invention is useful for cargo vehicle because in many cargo vehicles a driver's view in the rearward direction 
of the vehicle is blocked by a cargo compartment The application of the present invention is not limited to motor 
vehicles (including automobiles, large motor vehicles, such as buses, trucks, etc., and motor vehicles for cargoes). 

so The present invention is applicable to trains. 

(Embodiment 4) 

[0091] Figure 1 3A is a side view showing a train 37 which includes a surround surveillance system for a mobile body 
55 according to embodiment 4 of the present invention. Figure 1 3B is a plan view of the train 37 shown in Figure 13 A. In 
embodiment 4, the train 37 is a railroad train. 

[0092] In embodiment 4, as shown in Figures 13A and 13B, the omniazimuth visual sensors 4 of the surround sur- 
veillance system are each provided on the face of a car of the train 37 above a connection bridge. These omniazimuth 
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visual sensors 4 have 180° view fields in the running direction and in the direction opposite thereto, respectively. 
[0093] In embodiments 1 -4, the present invention is applied to a vehicle or a train. However, the present invention 
can be applied to at) types of mobile bodies, such as aeroplanes, ships, etc., regardless of whether such mobile bodies 
are manned/unmanned. 

[0094] Furthermore, the present invention is not limited to a body moving one place to another. When a surround 
surveillance system according to the present invention is mounted on a body which moves in the same place, the 
safety around the body when it is moving can readily be secured. 

[0095] In embodiments 1 -4, an optical system shown in Figure 3 is used as the optical system 4a which is capable 
of obtaining an image of 360° view field area the re around and capable of central projection transformation for the 
image. The present invention is not limited to such an optical system, but can use an optical system described in 
Japanese Laid -Open Publication No. 11-331654. 

[0096] As described hereinabove, according to the present invention, an omn azimuth visual sensor(s) is placed on 
an upper side, an end portion, etc., of a vehicle, whereby a driver's blind areas can be readily observed. With such-a 
system, the driver does not need to switch a plurality of cameras, to select one among these cameras for display on 
a display device, or to change the orientation of the camera, as in a conventional vehicle surveillance apparatus. Thus, 
when the driver starts to drive, when the motor vehicle turns right or left, or when the driver parks the motor vehicle in 
a carport or drives the vehicle out of the carport, the driver can check the safety around the vehicle and achieve safe 
driving. 

[0097] Furthermore, the driver can select a desired display image and change the display direction or the image 
size. Thus, for example, by switching a display when the vehicle moves rearward, the safety around the vehicle can 
be readily checked, whereby a contact accident(s) or the like can be prevented. 

[0098] Furthermore, h is possible to switch between a display of an image of the surroundings of the mobile body 
and a display of vehicle location. Thus, the space around the driver's seat is not narrowed, and manipulation of the 
system is not complicated as in the conventional system. 

[0099] Further still, a distance from an object around the mobile body, the relative velocity, a moving direction of the 
mobile body, etc., are determined. When the object comes into a predetermined area around the mobile body, the 
system can produce an aJarm. Thus, the safety check can be readily performed. 

[0100] various other modifications will be apparent to and can be readily made by those skilled in the art without 
departing from the scope and spirit of this invention. Accordingly, it is not intended that the scope of the claims appended 
hereto be limited to the description as set forth herein, but rather that the claims be broadly construed. 



Claims 

1. A surround surveillance system mounted on a mobile body for surveying surroundings around the mobile body, 
comprising an omniazimuth visual system, the omniazimuth visual system including: 

at least one omniazimuth visual sensor including an optical system capable of obtaining an image of 360° 
view field area therearound and capable of central projection transformation for the image, and an imaging 
section for converting the image obtained by the optical system into first image data; 

an image processor for transforming the first image data into second image data for a panoramic image and/ 
or for a perspective image; 

a display section for displaying the panoramic image and/or the perspective image based on the second image 
data; and 

a display control section for selecting and controlling the panoramic image and/or the perspective image. 

2. A surround surveillance system according to claim 1, wherein the display section displays the panoramic image 
and the perspective image at one time, or the display section selectively disptays one ol the panoramic image and 
the perspective image. 

3. A surround surveillance system according to claim 1 , wherein the display section simultaneously displays at least 
frontal, left, and right view field perspective images within the 360° view fie Id area based on the second image data. 

4. A surround surveillance system according to claim 3, wherein: 

the display control section selects one of the frontal, left, and right view field perspective images displayed by 
the display section; 

the image processor vertically/horizontally moves or scales- up/scales-down the view field perspective image 
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selected by the display control section according to an external operation; and 
the display section displays the moved or scaled-up/scaled-down image. 



5. A surround surveillance system according to claim 1 f wherein: 

5 

the display section includes a location display section for displaying a mobile body location image; and 

the display control section switches the display section between an image showing surroundings of the mobile 

body and the mobile body location image. 

10 6. A surround surveillance system according to claim 1 . wherein the mobile body is a motor vehicle. 

7. A surround surveillance system according to claim 6, wherein the at least one omniazimuth visual sensor is placed 
on a roof of the motor vehicle. 



'5 8. A surround surveillance system according to claim 6, wherein: 

the at least one omniazimuth visual sensor includes first and second omniazimuth visual sensors; 
the first omniazimuth visual sensor is placed on a front bumper of the motor vehicle; and 
the second omniazimuth visual sensor is placed on a rear bumper of the motor vehicle. 

20 

9. A surround surveillance system according to claim 8, wherein: 

the first omniazimuth visual sensor is placed on a left or right corner of the front bumper; and 
the second omniazimuth visual sensor is placed at a diagonal position on the rear bumper with respect to the 
25 first omniazimuth visual sensor. 

10. A surround. surveillance system according to claim 1 , wherein the mobile body is a train. 

11. A surround surveillance system according to claim 1, further comprising: 

30 

means for determining a distance between the mobile body and an object around the mobile body, a relative 
velocity of the object with respect to the mobile body, and a moving direction of the object based on a signal 
of the image data from the at least one omniazimuth visual sensor and a velocity signal from the mobile body; 
and 

35 alarm ing means for producing alarming information when the object comes into a predetermined area around 

the mobile body. 

12. A surround surveillance systBm, comprising: 

40 an omniazimuth visual sensor including an optical system capable of obtaining an image of 360° view field 

area therearound and capable of central projection transformation for the image, and an imaging section for 
converting the image obtained by the optical system into first image data; 

an image processor for transforming the first image data into second image data for a panoramic image and/ 
or for a perspective image; 

45 a display section for displaying the panoramic image and/or the perspective image based on the second image 

data; and 

a display control section for selecting and controlling the panoramic image and/or the perspective image. 

13. A mobile body, comprising the surround surveillance system of claim 12. 

so 

14. A motor vehicle, comprising the surround surveillance system of claim 12. 

15. A train, comprising the surround surveillance system of claim 12. 

55 
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